In general, virtualization is a technique that provides an ability to host two or more operating systems concurrently on the same computing device. Currently, virtualization architectures feature hardware virtualization capabilities that operate in accordance with two processor modes: host mode and guest mode. A virtual machine runs in guest mode, where a processor of the computing device switches from host mode to guest mode for execution of software components associated with the virtual machine (VM entry). Similarly, the processor may switch from guest mode to host mode when operations of the virtual machine are paused or stopped (VM exit).
A virtual machine (VM) is a software abstraction that operates like a physical (real) computing device having a particular guest operating system (OS). Each VM includes an operating system (OS) kernel (sometimes referred to as a “guest kernel”) that operates in a most privileged guest mode (guest kernel mode, ring-0). A guest software application executes in a lesser privileged operating mode (guest user mode, ring-3).
As the guest software application executes and requires certain resources, the guest software application accesses an Application Programming Interface (API), which invokes a system call function (sometimes referred to as a “syscall”) operating within the guest kernel. In response to the syscall, the guest kernel operates as a service provider by facilitating communications between the guest software application and one or more resources associated with the syscall. Examples of the resources may include, but are not limited or restricted to virtual resources, such as a particular virtual driver or certain virtual system hardware implemented by software components in a host space (e.g., virtual central processing unit “vCPU” or virtual disk) that are configured to operate in a similar manner as corresponding physical components (e.g., physical CPU or hard disk) or directly mapped to a physical resource. As a result, the guest system software, when executed, controls execution and allocation of virtual resources so that the VM operates in a manner consistent to operations of the physical computing device.
With the emergence of hardware support for full virtualization in an increased number of hardware processor architectures, new virtualization (software) architectures have emerged. One such virtualization architecture involves adding a software abstraction layer, sometimes referred to as a “virtualization layer,” between the physical hardware and the virtual machine. The virtualization layer runs in host mode. It consists of an OS kernel (sometimes referred to as a “hypervisor”) and multiple host applications that operate under the control of the OS kernel.
Conventionally, the detection of an exploit usually involves monitoring the usage of System APIs accessed by the guest software application operating with the virtual machine. This is achieved by intercepting (sometimes referred to as “hooking”) an API call in the guest user mode prior to entry into the guest kernel mode. The analysis as to whether the API call is associated with a malicious attack may be conducted in the guest user mode, and thereafter, control may be transferred back to service the API function being called. However, there are exploits designed to detect an API hook, and in response, attempts to bypass the API hook by advancing a memory pointer directed to the “hooked” instruction associated with the API call a few bytes. The bypass attempt, referred to as “hook hopping,” is an attempt by the exploit to execute the original API function.
Previously, to avoid hook hopping, efforts have been made to migrate exploit detection, one type of malware detection, into the guest kernel through a technique known as “kernel hooking”. Kernel hooking is the process of modifying the guest OS kernel to alter its behavior or capture certain events. Previously, security vendors relied on kernel hooking to implement antivirus services, protecting the OS and its applications by intercepting and blocking potentially malicious actions or processes. However, in an attempt to strengthen protection of the guest kernel and combat the increased presence of rootkit attacks, some software companies modified their operating systems to monitor kernel code as well as system resources used by the guest kernel and initiate an automatic shutdown of the computing device upon detecting unauthorized kernel patching. One example of this modification is PatchGuard™ for Microsoft® Windows® operating systems.
While the prevention of kernel patching may have reduced some rootkit infections, it precludes security providers from providing more robust protection against malicious attacks to the guest kernel by handling exploit detection checks in the guest kernel.